Skip to content

Keep non-ASCII unescaped in wire serialization - #1964

Open
KarataevDmitry wants to merge 3 commits into
modelcontextprotocol:mainfrom
KarataevDmitry:relaxed-json-escaping
Open

Keep non-ASCII unescaped in wire serialization#1964
KarataevDmitry wants to merge 3 commits into
modelcontextprotocol:mainfrom
KarataevDmitry:relaxed-json-escaping

Conversation

@KarataevDmitry

@KarataevDmitry KarataevDmitry commented Oct 26, 2026 •
edited
Loading

Copy link
Copy Markdown

Summary

Wire serialization (stdio server/client, HTTP client, SSE writer) goes through the frozen
source-generated JsonContext.Default.Options, whose conservative encoder escapes every
non-ASCII character into a six-character sequence - a Cyrillic/CJK tool payload costs x2-3
tokens when the consumer is an LLM, not an HTML embedder.

Escaping is semantically neutral in JSON: clients parse "б" and "б" into the identical
string. MEAI (AIJsonUtilities.DefaultOptions) and ASP.NET Core already default to
JavaScriptEncoder.UnsafeRelaxedJsonEscaping; with this change MCP matches them.

Changes

  • McpJsonUtilities.CreateDefaultOptions: set the relaxed encoder (next to the existing
    converter/resolver customization, before MakeReadOnly).
  • Serialize wire messages via McpJsonUtilities.DefaultOptions.GetTypeInfo<JsonRpcMessage>()
    instead of JsonContext.Default.JsonRpcMessage: StreamServerTransport, StreamClientSessionTransport,
    McpHttpClient (both NET and non-NET paths), McpSseEventWriterExtensions.
    DefaultOptions already exists precisely to customize the context defaults, and
    [JsonSourceGenerationOptions] cannot express an encoder, so the hand-written options are the
    right single source for the wire.
  • Propagate the same encoder into the SSE Utf8JsonWriter (JsonWriterOptions.Encoder).
  • Log-only serialization sites left on the generated context (not wire).

Measurement

Real-world check (our AgentNotes KB MCP server, Russian/Chinese corpus, read_hot_context tool,
identical JSON-RPC exchange, wire bytes measured raw):

before after
wire bytes 21 295 8 925 (-58 %)

Test

  • New McpJsonUtilitiesTests.DefaultOptions_KeepsNonAsciiUnescaped asserts a Cyrillic/CJK string
    round-trips unescaped through DefaultOptions on net8.0/net9.0/net10.0/net472.
  • Full test suite: 3358 passed on each .NET TFM. Two DockerEverythingServerTests integration
    tests fail identically on a clean checkout: the pinned third-party image
    tzolov/mcp-everything-server:v1 (up to date as of this run) no longer exposes the
    trigger-sampling-request tool the test calls. Pre-existing environment drift, unrelated to
    this change.

Addresses #626.

@dfedoryshchev

dfedoryshchev commented Oct 27, 2026

Copy link
Copy Markdown
Contributor

You missed one writer: src/ModelContextProtocol.AspNetCore/SseEventStreamReaderExtensions.cs:45 still does new Utf8JsonWriter(writer) with no options. When JsonSerializer.Serialize gets a Utf8JsonWriter, the writer's encoder wins over JsonSerializerOptions.Encoder, which is why McpSseEventWriterExtensions needed its own writer options here.

That writer serves resumability replay (StreamableHttpHandler.cs:278 and :315), so with an event store a reconnecting client gets replayed events escaped and live ones unescaped.

The new test only covers SerializeToUtf8Bytes; formatting an SseItem through each SSE writer would catch this.

@KarataevDmitry

KarataevDmitry commented Oct 27, 2026

Copy link
Copy Markdown
Author

@dfedoryshchev Good catch — you're right on both counts.

Fixed in 7a6472e: SseEventStreamReaderExtensions now passes the same JsonWriterOptions as McpSseEventWriterExtensions, so replayed events match live ones for reconnecting clients.

The new test now covers both writers: ResumedStreamData_KeepsNonAsciiUnescaped (in ResumabilityIntegrationTestsBase) reads progress notifications from the live POST response stream and then again via a Last-Event-ID replay. It fails without the writer fix (replayed Cyrillic arrives escaped) and passes with it, in both the in-memory and distributed-cache store suites.

Also rebased onto current main (#1934 restructured McpJsonUtilities).

@KarataevDmitry
Keep non-ASCII unescaped in wire serialization
8b63bb3 
MCP payloads are read by LLMs, not embedded into HTML. The conservative
default encoder escapes every non-ASCII char into a six-character
sequence, costing x2-3 tokens on Cyrillic/CJK tool results. Escaping is
semantically neutral in JSON; MEAI (AIJsonUtilities) and ASP.NET Core
already default to JavaScriptEncoder.UnsafeRelaxedJsonEscaping.

- Set the relaxed encoder on McpJsonUtilities.DefaultOptions
- Serialize wire messages (stdio server/client, HTTP client, SSE writer)
  via DefaultOptions instead of the frozen generated JsonContext options
- Propagate the encoder into the SSE Utf8JsonWriter

Addresses modelcontextprotocol#626.

Signed-off-by: Каратаев Дмитрий <dkarataev1990@gmail.com>
@KarataevDmitry
Propagate the relaxed encoder into the resumability replay writer
028096a 
The replay path (SseEventStreamReaderExtensions.CopyToAsync) created its
Utf8JsonWriter without options, and a writer's encoder overrides
JsonSerializerOptions.Encoder - so a reconnecting client with an event
store received replayed events escaped while live ones were unescaped.

Pass the same JsonWriterOptions as McpSseEventWriterExtensions, and cover
both writers with an integration test that reads progress notifications
from the live POST response stream and from a Last-Event-ID replay.

Signed-off-by: Каратаев Дмитрий <dkarataev1990@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@KarataevDmitry @dfedoryshchev